feat(logs): max template size limit + json_as_raw raw bypass#49552
Draft
DDuongNguyen wants to merge 2 commits into
Draft
feat(logs): max template size limit + json_as_raw raw bypass#49552DDuongNguyen wants to merge 2 commits into
DDuongNguyen wants to merge 2 commits into
Conversation
This was referenced Apr 17, 2026
Contributor
Author
Contributor
Go Package Import DifferencesBaseline: 228994b
|
Contributor
Files inventory check summaryFile checks results against ancestor a9034095: Results for datadog-agent_7.79.0~devel.git.459.3865993.pipeline.108330740-1_amd64.deb:Detected file changes:
|
DDuongNguyen
force-pushed
the
yoon/max-template-raw-bypass
branch
from
f825ba6 to
3865993
Compare
This was referenced Apr 20, 2026
Draft
Draft
Draft
This was referenced Apr 28, 2026
Draft
Draft
Draft
This was referenced May 5, 2026
Draft
Draft
Two safety valves that prevent stateful encoding state bloat: **logs_config.patterns.max_template_bytes (default: 1024)** Rejects logs whose raw token content exceeds this byte threshold, sending them as RawLog datums instead of creating pattern state. Fixes snapshot bloat from huge one-off logs (e.g. AWS instance metadata dumps ~900KB each → ~8MB snapshot overhead from 9 patterns). Returns PatternTooLarge from ClusterManager.Add so the caller never dereferences a nil pattern. **logs_config.patterns.json_as_raw (default: false)** When true, JSON logs bypass stateful encoding entirely (no preprocessing, no tokenization, no clustering) and are sent as RawLog datums. Eliminates PreprocessJSON + tokenization + clustering cost for JSON-heavy workloads. Trade-off: no pattern compression; transport-level compression still applies. Useful for A/B testing CPU vs wire efficiency in staging. Both features route to the existing sendRawLog path in both the batch pipeline (processBatch) and single-message path (processMessage). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
UseNumber in json.Unmarshal so numbers stay as json.Number (string) instead of float64. Without this, integers larger than 2^53 (trace IDs, span IDs, snowflake IDs) silently lose precision through the unmarshal/marshal round-trip — e.g. 9999999999999999 becomes 10000000000000000.
TheSafo
force-pushed
the
jsaf/evict-correctly
branch
from
6a19a73 to
f229313
Compare
TheSafo
force-pushed
the
yoon/max-template-raw-bypass
branch
from
0bb9291 to
ed5c750
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Motivation
Describe how you validated your changes
Additional Notes